System and method for enabling a collaborative desktop environment

ABSTRACT

Described herein are systems and methods for enabling a collaborative remote desktop environment. The system includes a computing device and a first application instance that has an application state associated therewith. The first application instance includes, or is associated with, a current state component and application data/data files. The system further includes an application launcher that is used to instantiate a second application instance operating either on the same or on a different computing device. The second application instance similarly has an application state associated therewith and is associated with the application launcher. Upon receiving a request from the second user to interact with the first application instance, the application state and the application data/data files are communicated to the application launcher, and the application launcher instantiates the second application instance so that its state is substantially identical to that of the first application instance.

CLAIM OF PRIORITY

This application is related to and claims priority from co-pending U.S.Utility patent application Ser. No. 13/213,025 entitled SYSTEM ANDMETHOD FOR A COLLABORATIVE DESKTOP ENVIRONMENT filed on Aug. 18, 2011,which is related to and claims the benefit of priority to U.S.Provisional patent application titled SYSTEM AND MACHINE FOR ACOLLABORATIVE REMOTE DESKTOP ENVIRONMENT, Application No. 61/402,477filed Aug. 31, 2010, all of which have common inventor AndersNancke-Krogh, and both of which are incorporated by reference herein intheir entirety.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

Embodiments of the present invention are generally related to computersystems, and user-computer interaction and collaboration, and areparticularly related to systems and methods for enabling a collaborativedesktop environment in a computer, media, or other environment.

BACKGROUND

A remote desktop system generally allows a user at a client device(i.e., a remote

user) to view an information display or desktop of another user's device(i.e., a host user). The technology behind many of today's remotedesktop systems consists of presenting a live feed of one user'sinformation display or desktop to one or more other users' informationdisplay or desktop, and to allow the other users to interact with anapplication at the host user's device through a remote desktop window.Such systems are convenient and cost effective for allowing a remoteuser to, e.g., help a user at a host device with administration of thehost user's device. For example, the remote desktop system can renderthe host user's information display on the remote user's display deviceas a remote desktop window, so that any interactions made by either theremote user or the host user is executed by the applications running onthe host user's computer.

Remote desktop systems have also been used in net-meeting orpresentation-like environments. In such environments, a user at a hostdevice can present information about an application at the host deviceto remote users, while simultaneously restricting the remote users frominteracting with the host application. Variations of such systems allowthe host user to control which, if any, of the remote users can controland/or interact with the host application. Additionally, some remotedesktop systems allow multiple remote users to connect with and sharecontrol of a host application.

However, to date, many remote desktop systems have focused onapplications running only on the host user's device, and provide littlein the area of collaboration. For example, where an application isrunning on a host user's device, any interaction made by a remote userthrough the remote desktop system will affect all of the other usersviewing that host application. This is particularly unsatisfactory inapplications such as web browsing, where, e.g., a host user may retrievea web page, and when the host user has finished reading the web page,the host user will navigate to another page by selecting a hyperlink.Remote users, who may be viewing the same web page through a remotedesktop window, may be reading slower or faster than the host user.Those remote users who finish reading will be required to wait for thehost user, but might want to explore the content behind one of thehyperlinks on the page. However, if a client user remotely interactingwith the host application selects a link (assuming the host user hasgranted them permission to interact with the host's web browser), thenthe host application will navigate to this new link and the host userwill be interrupted, as will any other remote readers who have notfinished reading the content. Alternatively, if the host user decidesnot to share control of their browser application, the host user mightbe asked to copy the uniform resource locator (URL), and send it (e.g.,via email or instant message) to the remote users, so that therequesting remote user can perform their own simultaneous browsing whilekeeping the shared desktop window open. This will similarly interruptthe host user, and delay things for all of the users. These are thegeneral areas that embodiments of the invention are intended to address.

SUMMARY

Described herein are systems and methods for enabling a collaborativeremote desktop environment. In accordance with an embodiment, the systemincludes a computing device and a first application instance (e.g., aweb browser or other application) that has an application stateassociated therewith. The first application instance includes, or isassociated with, an input handler, an application logic component, anoutput handler, a current state component, and application data/datafiles. When a second user interacts with the first application instance,requests (e.g., interaction requests) from the second user are receivedat an operating system remote input handler at the computing device. Theoperating system remote input handler communicates the interactionrequests to the current state component; the current state componentcommunicates the application state and the application data/data filesto the operating system remote input handler. In accordance with anembodiment, the system further includes an application launcherdesignated to the second user. The application launcher is used toinstantiate a second application instance (e.g., another web browser oranother application), operating either on the same or on a differentcomputing device. The second application instance has an applicationstate associated therewith, and similarly includes an input handler, anapplication logic component, an output handler, a current statecomponent, and application data/data files. The second applicationinstance is associated with the application launcher, wherein theapplication launcher can receive the application state data and theapplication data/data files from the first application, via theoperating system remote input handler, and instantiates the secondapplication instance so that its state is substantially identical tothat of the first application instance. Upon instantiating the secondapplication instance, the original interaction request made by thesecond user is communicated to the input handler of the secondapplication instance for processing.

For example, in accordance with an embodiment, a first user can use,e.g., a multi-touch enabled desktop computer with a browser window, tobrowse the web within a browsing session. A second user, who wants tocollaborate with the first user, in their own browsing session andstarting from a substantially similar position as the first user, caninstantiate a new browser window which uses the same uniform resourcelocator (URL), content, zoom and scroll bar position, etc., as theoriginal browser window, i.e. the new browser window includessubstantially the same state as the original browser window and browsingsession.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an illustration of a high-level view of a typical computingdomain or

environment, in accordance with an embodiment.

FIG. 2 illustrates a more-detailed view of the computing domain orenvironment in FIG. 1, including an application and operating system, inaccordance with an embodiment.

FIG. 3 illustrates a more-detailed view of the computing domain orenvironment in FIG. 2, including an application input handler,application logic, application output handler, and application data inaccordance with an embodiment.

FIG. 4 shows an illustration of how a computing domain or environmentcan be used to enable a remote desktop system, in accordance with anembodiment.

FIG. 5 illustrates how a computing domain or environment, such as thatshown in FIG. 4, can instead utilize an application's current state, toenable a collaborative desktop environment, in accordance with anembodiment.

FIG. 6 shows a flow chart of a method in accordance with an embodiment,for providing a collaborative remote desktop system.

FIG. 7 shows an illustration of an exemplary system for two-waycollaboration, in accordance with an embodiment.

-   -   3

FIG. 8 shows a flow chart of a method in accordance with an embodiment,for providing a two-way collaboration system.

FIGS. 9A-9D show an illustration of how the system can be used toprovide a shared desktop implementation, in accordance with anembodiment.

FIGS. 10A-10C show an illustration of how the system can be used toprovide a shared group implementation, in accordance with an embodiment.

As described above, the technology behind many of today's remote desktopsystems consists of presenting a live feed of one user's informationdisplay or desktop to one or more other users' information display ordesktop, and to allow the other users to interact with an application ata host user's device through a remote desktop window. Such systems areconvenient and cost effective for allowing a remote user to, e.g., helpa user at a host device with administration of the host user's device.However, to date, many remote desktop systems have focused onapplications running only on the host device, and provide little in thearea of collaboration.

To address this, described herein are systems and methods for enabling acollaborative remote desktop environment. In accordance with anembodiment, the system includes a computing device and a firstapplication instance (e.g., a web browser or other application) that hasan application state associated therewith. The first applicationinstance includes, or is associated with, an input handler, anapplication logic component, an output handler, a current statecomponent, and application data/data files. When a second user interactswith the first application instance, requests (e.g., interactionrequests) from the second user are received at an operating systemremote input handler at the computing device. The operating systemremote input handler communicates the interaction requests to thecurrent state component; the current state component communicates theapplication state and the application data/data files to the operatingsystem remote input handler. In accordance with an embodiment, thesystem further includes an application launcher designated to the seconduser. The application launcher is used to instantiate a secondapplication instance (e.g., another web browser or another application),operating either on the same or on a different computing device. Thesecond application instance has an application state associatedtherewith, and similarly includes an input handler, an application logiccomponent, an output handler, a current state component, and applicationdata/data files. The second application instance is associated with theapplication launcher, wherein the application launcher can receive theapplication state data and the application data/data files from thefirst application, via the operating system remote input handler, andinstantiate the second application instance so that its state issubstantially identical to that of the first application instance. Uponinstantiating the second application instance, the original interactionrequest made by the second user is communicated to the input handler ofthe second application instance for processing.

For example, in accordance with an embodiment, a first user can use,e.g., a multi-touch enabled desktop computer with a browser window, tobrowse the web within a browsing session. A second user, who wants tocollaborate with the first user, in their own browsing session andstarting from a substantially similar position as the first user, caninstantiate a new browser window which uses the same uniform resourcelocator (URL), content, zoom and scroll bar position, etc., as theoriginal browser window, i.e. the new browser window includessubstantially the same state as the original browser window and browsingsession.

GLOSSARY OF TERMS

In accordance with an embodiment, the following terms are used herein:

An “information display” refers to the visual output of a computer thatcan be controlled by one or more persons or users, and is sometimesreferred to as the “desktop” of the computer. In accordance with anembodiment, the information display can be rendered over one or morephysical display devices. In accordance with other embodiments, theinformation display can be provided such that it covers only a portionof a display device—sometimes referred to as a “split-screen” system.

A “computer” refers to an electronic or computing device that takesinput from one user and renders the derived output to an informationdisplay. A computer can be implemented as physical hardware or virtualhardware in a physical hardware computing device.

A “computing device” refers to a physical mobile or stationary machinethat hosts one or more computers.

A “display device” refers to a physical display that renders the outputfrom one computing device.

A “remote desktop” refers to a system where the information display of auser's computer is rendered in a window on a plurality of other users'information displays.

A “window” refers to a visible output of an application as rendered onan information display. Depending of the capability of the computerused, several windows can be displayed simultaneously on the informationdisplay, and the windows can be layered, moved and resized.

An “application” refers to a software program that is previouslyinstalled and now executed by a computer. The computer controls whichinputs from the user are directed to the application, and the computercontrols which outputs the application is allowed to render on thedisplay device. Depending of the capability of the computer, one orseveral applications can be installed simultaneously and/or runningsimultaneously.

An “application state” comprises one or more of the application, thecontent file(s) the application is using and/or writing, and/or allinputs made by the user of this application since the user started usingthe application for a particular launch of the application (e.g., allinput from the opening of the application). In accordance with anembodiment, the application state provided by a first applicationinstance must be adequate and detailed enough to enable another computeror application instance to reach the same (target) application state, oran application state as close as possible or substantially identical,synchronized, or otherwise similar to the target application state.

In accordance with an embodiment, an “msync”, sometimes referred to asan “online-media-sync” is the process by which the system enables anapplication or user to copy the state of an application session (couldbe a browser) associated with another application or user, so that theuser can thereafter continue within their own copy of the applicationsession, starting from a substantially identical, synchronized orotherwise similar position and state, without interrupting the otheruser.

INTRODUCTION

FIG. 1 shows an illustration of a high-level view of a typical computingdomain or

environment, in accordance with an embodiment. As shown in FIG. 1, thesystem includes a user 102, a computer 106, and an information displayor desktop (e.g., a computer monitor) 110. The user can interact withthe computer by providing input 104 to the computer. The various typesof input methods that can be used include, but are not limited to, mouseinput (e.g., left, right, double clicks using the mouse), touch input(e.g., touch and dragging with one or more fingers at a monitor ortouchpad), and keyboard input. The computer receives the input and makescalculations to produce an output 108 (e.g., a visual output, such as adesktop display) based on the input received at the computer and thecurrently running applications, along with the current state and currentfiles used and/or controlled by the applications. The visual output canbe a visual representation that is displayed on the information display.

FIG. 2 illustrates a more-detailed view of the computing domain orenvironment in FIG. 1, including an application and operating system, inaccordance with an embodiment. As shown in FIG. 2, the system includes auser 202, a computer 206, and an information display 210. The computerincludes an Operating System (OS) input handler 212, OS output handler214, and one or more applications. The OS input handler receives input204 from the user, and forwards that input to an application 208 forfurther processing. The OS output handler receives the requested visualoutput from each of the currently running applications on the computer,and determines the portion of the visual output received from each ofthe applications that will be rendered on the information display. TheOS output handler outputs the visual output 209 to the informationdisplay. In the event there is more than one visual output from theapplications being partly on top of each other, a partial rendering ofthose visual outputs can be displayed.

FIG. 3 illustrates a more-detailed view of the computing domain orenvironment in FIG. 2, including an application input handler,application logic, application output handler, and application data inaccordance with an embodiment. As shown in FIG. 3, the system includes auser 302, a computer 306, and an information display 310. An application308 running on the computer includes an application input handler 316,application logic 318, application output handler 320, and applicationdata/data files 322. Input 304 is received by the application inputhandler from an OS input handler 312. The application input handlercommunicates the input to the application logic component, which uponprocessing the input updates the application state, and the applicationdata/data files used by the application.

In accordance with an embodiment, described in further detail below, theapplication state must be adequate and detailed enough to enable anothercomputer or application instance to reach the same (target) applicationstate, or an application state as close as possible or substantiallyidentical, synchronized, or otherwise similar to the target applicationstate. To this end, in accordance with an embodiment, the applicationstate can include an application start condition, a data file startcondition, and the sequence and timing of all inputs received by the OSinput handler. The application converts the current state to a visualoutput to be displayed on an information display. An OS output handler314 receives the converted current state from the application outputhandler, and decides how much of the visual output will be displayed onthe information display, where on the information display the visualoutput will be located, and outputs 309 the visual output to theinformation display.

FIG. 4 shows an illustration of how a computing domain or environmentcan be used to enable a remote desktop system, in accordance with anembodiment. As shown in FIG. 4, in a “classical” remote desktopenvironment, a host user 401 can interact with a client user 402 about,e.g., content discovery. The host user can initiate the discovery bystarting a first application 408 (e.g., a host application) at a firstdevice 406 (e.g., a host computer). The host application can be, forexample, a web browser or another application. The host user caninteract with the host application by providing input 441 to thecomputer. As described above, the various types of input methods thatcan be used include, but are not limited to, mouse input (e.g., left,right, double clicks using the mouse), touch input (e.g., touch anddragging with one or more fingers at a monitor or touchpad), andkeyboard input. The client user creates a remote desktop connectionusing a remote desktop client application 409 at the client user'scomputer 407. The host computer initializes an OS remote input handler436 and an OS remote output handler 440. The OS remote input handlerreceives input (e.g., a request to interact with the host application)from an input handler 426 at the remote desktop client application,where the input can be from the client user.

When the client user interacts with the remote desktop clientapplication, a command to interact 442 (i.e., an interaction command) isreceived at, then communicated from an OS client input handler 413 tothe input handler at the remote desktop client application. The inputhandler communicates the interaction command to the OS remote inputhandler on the host computer, which in turn communicates the interactioncommand to an OS host input handler 412.

The interaction command is treated as though it were sent by the hostuser. For example, once the interaction command is received by the hostapplication at an input handler 416, the application responds accordingto the interaction command, not knowing if the input originated from thehost or client user. The input handler at the host applicationcommunicates the interaction command to an application logic component418, where the host computer uses data in the application logiccomponent and application data/data files 422 to create a visual outputto be displayed on an information display 410 (e.g., a host informationdisplay). For example, an OS output handler 414 can receive the visualoutput from an application output handler 420, and decide how much ofthe entire window will be displayed and where on the information displaythe window will be located before outputting 443 the visual output tothe information display.

In accordance with an embodiment, the OS output handler communicates thevisual display to the OS remote output handler, which in turncommunicates the visual output to an output handler 430 at the remotedesktop client application. The output handler communicates the visualoutput to an OS output handler 415 to be communicated 444 to aninformation display 411 (e.g., client information display). Inaccordance with an embodiment, the information display renders a windowshowing a current (e.g., live) representation of the information displayof the host user's computer. Inputs received at the host applicationresult in a direct update of the host application and the host user'sinformation display. These updates are communicated from the OS remoteoutput handler to the remote desktop output handler at the remotedesktop application, and the client user's information display isupdated in real time with the same visual representation as displayed atthe host user's information display. The host information display andthe client information user display are updated based on the interactioncommand. For example, the host application is displayed as a windowwithin the host information display, and the viewable area of the hostinformation display is displayed within a window on the clientinformation display.

Two-Way Collaborative Sharing

FIG. 5 illustrates how a computing domain or environment, such as thatshown in FIG. 4, can instead utilize an application's current state, toenable a collaborative desktop environment, in accordance with anembodiment. As shown in FIG. 5, interactions made by a user (e.g., aclient user) through a remote desktop window on an application (e.g., ahost application) can result in the host application being opened on theclient user's computer. In accordance with an embodiment, the clientuser can have two applications running, a remote desktop window wherethe client user follows another user's (e.g., a host user) activities,and a new instance of the host application running locally on the clientuser's computer, but in a different state. User activity on the localapplication made by the client user only effects the local application.Activity made by the host user on the host application will result in achange on the host applications, and an updated information display inthe client's remote desktop windows.

Although in the example shown in FIG. 5, a host computer having a hostapplication is described, wherein the host application is accessed by aclient user, in accordance with other embodiments, the system can resideon a single shared computing device. For example, the shared computingdevice can have a first application instance, wherein the user activitymade by a first user at the shared computing device instantiates asecond application on the shared computing device having a statesubstantially identical to that of the first application, for use by asecond user. Additionally, as described above, the labels “host” and“client” are used herein to differentiate two instances of anapplication (or device), but these labels can be interchanged.

As shown in FIG. 5, a user 502 (e.g., a client user) can interact with aremote desktop application 509 at a computer 507 (e.g., a clientcomputer). The client user can, e.g., interact with an application 508(e.g., host application) at a computer 506 (e.g., host computer).Although in this example a host user and client user are described, anyplurality of users can collaborate. In accordance with an embodiment,the host computer includes an application having an input handler 516, alogic component 518, an output handler 520 and application data/datafiles 522. An interaction input/request 570 from a user 501 (e.g., hostuser) is received by the application input handler from an OS inputhandler 512. The application input handler communicates the input (i.e.,interaction input/request) to the application logic component whichupdates the application state and updates the application data/datafiles used by the application. The host application converts the currentstate to a visual output to be displayed on an information display 510.For example, an OS output handler 514 receives the visual output fromthe application output handler. The OS output handler decides how muchof the entire window will be displayed, where on the information displaythe window will be located, and communicates the visual output 572 tothe information display. The OS output handler also communicates thevisual output to an OS remote output handler 540, which in turncommunicates the visual output to an output handler 530 at the remotedesktop client application for display at an information display 511(e.g., a client information display).

In accordance with an embodiment, the remote desktop client applicationincludes an input handler 526 and an output handler. The input handlerat the remote desktop communicates 564 the interaction request to an OSremote input handler 536, which requests 565 a complete record of thehost application's current state from a current state component 523 atthe host application. In accordance with an embodiment, the currentstate component collects application data/data files, which can includemetadata describing how to start the host application on the clientuser's computer, and metadata to instantiate a local instance of thehost application on the client computer having the same state as thehost application. For example, the application data/data files caninclude instructions describing how to download and install the hostapplication, the application settings such as menus and options that areenabled on the host application, the privacy setting (e.g., shared orprivate) of the file(s) controlled by the host application, the location(e.g., a URL or network sharing information) for the file(s) controlledby the host application, the current configuration information of thehost application, such as scroll location of all scrollable andconfigurable sections of the host application, application sessioninformation such as cookie files (e.g., when the host application is aweb browser), and information about actionable sections of the hostapplication such as a URL behind a link on the host application.Alternatively, if a cloud-based system is used to maintain theapplication data/data files, the current state component can refer to alink to the location of the application data/data files in thecloud-based system. The current state information, local files, and ifapplicable, the link to the cloud-based system having the applicationdata/data files are returned 566 to the OS remote input handler.

In accordance with an embodiment, the OS remote input handler at thehost computer communicates an application launch request 567 to anapplication launcher 504. The application launcher receives theapplication data/data files, in addition to the application/launchinformation. In accordance with an embodiment, if the host applicationis not installed on client user's computer, the application launcherinitiates a download and install of a compatible host application forthe client user's computer. If the host application is a “premiumapplication” (e.g., an application that requires some form of payment),then the application launcher requests payment for the application. Onceinstalled, the application launcher instantiates 568 an instance of thehost application as local application 538 on the client computer.

The local application includes an input handler 546, application logic548, an output handler 550, a current state component 553 andapplication data/data files 552. Once the local application is started,application launcher communicates 569 the application data/data files tothe current state component, and adjusts the local application's currentstate to the same state as the host application's state using thecurrent state information (i.e., the application data/data files) of thehost application. For example, the local application data/data fileswill be updated and all internal state variables will be set toreplicate the host applications state variables, including link tocloud-based files if applicable. The cursor and application position andsize can also be set to the same

position as in the host application.

In accordance with an embodiment, the application launcher communicates574 the original interaction input/request made by the client user onthe remote desktop application to the input handler of the localapplication. For example, an input 571 received at an OS input handler513 is communicated to the input handler 546 on the local application.The local application converts the current state to a visual output tobe displayed on the information display. An OS output handler 515receives the visual output from the output handler at the localapplication. The OS output handler decides how much of the entire windowwill be displayed, where on an information display the window will belocated, and communicates the visual output 573 to the informationdisplay.

FIG. 6 shows a flow chart of a method in accordance with an embodiment,for providing two-way collaboration in a collaborative desktop system.As shown in FIG. 6, at step 602, a user (e.g., a client user) interactswith an application (e.g., a host application). The client user caninteract with the host application using a remote desktop clientapplication. The interaction can include, e.g., selecting a web link onthe host application. At step 604, a remote desktop input handler at ahost computer forwards an interaction request to an OS remote inputhandler on the host computer. Although in this example a host user andclient user are described, any plurality of users can collaborate.

At step 606, upon receiving the interaction request, the OS remote inputhandler obtains a complete record of the host application's currentstate from a current state component. In accordance with an embodiment,the current state component collects host application data/data files,such as current application state information. In accordance with anembodiment, the application data/data files can include metadatadescribing how to start the host application on the client user'scomputer, and metadata to instantiate a local instance of the hostapplication on the client computer having the same state as the hostapplication. At step 608, the OS remote input handler communicates anapplication launch request to an application launcher at the clientcomputer. The current state information, local files, and if applicable,a link to the cloud-based system having the current application statefiles are used to launch an instance of the host application on theclient user's computer. At step 610, the application launcher receivesthe application launch request, the application state information, inaddition to the host application installation/launch information, tolaunch an instance of the host application on the client user'scomputer.

In accordance with an embodiment, the application launcher determineswhether the host application is installed on the client user's computer.If the host application is not installed on the client's computer, thenan instance of the host application is installed on the client user'scomputer. If the host application is a premium application (e.g., anapplication that requires payment), then the application launcherrequests payment. At step 612, an instance of the host application isinstalled and instantiated on the client computer. At step 614, uponinstalling and instantiating the application instance, the applicationlauncher adjusts the application instances current state to the samestate as the host application's state. Once the state is set for theapplication instance, the application launcher communicates the originalinteraction request made by the client user to the input handler of theapplication instance. The application instance then operates independentfrom the application at the host computer which it was instantiatedfrom.

FIG. 7 shows an illustration of an exemplary system for two-waycollaboration, in accordance with an embodiment. Although in the exampleshown in FIG. 7, a first user and a second user are described, anyplurality of users can collaborate. Additionally, although the userswill be referred to as a first user and a second user, either user canbe an active user or a passive user. In accordance with an embodiment,an active user shares their information display to another user. Anactive user can become a passive user if they discontinue sharing theirinformation display, and a passive user can become an active user bysharing their information display. For example, a passive user can havea remote desktop application running to collaborate with an activeuser's computer. The passive user can become an active user when thepassive user shares their information display to other users.

As shown in FIG. 7, two users are in a two-way collaborative remotedesktop setup. As described above, although in this example a firstcomputer having a first application is described, where the firstapplication is accessed by a second user, in accordance with otherembodiments, the system can reside on a single shared computing device.For example, the shared computing device can have a first applicationinstance, where user activity made by a different user at the sharedcomputing device instantiates a second application on the sharedcomputing device having a state substantially identical to that of thefirst application. As further shown in FIG. 7, user 701 (e.g., firstuser) is running an application 708 (e.g., a local application) whichthe user interacts with, and a remote desktop client application 750showing a second user's 702 interactions with an application 709 (e.g.,a local application). The second user is running the application whichthe user interacts with, and a remote desktop client application 751showing the first user's interactions with their local application.

In accordance with an embodiment, changes made by the second user ontheir local application are executed locally on their computer. Theresult of the changes is visible on information display 711, in additionto a remote desktop window on the first user's information display 710.For example, computer 707 includes the local application and the remotedesktop client application. The remote desktop client applicationincludes an input handler 726 and an output handler 730. Input 781 fromthe second user is received at an OS input handler 713, whichcommunicates the input to the local application. The local applicationprocesses the input and produces a visual output to be displayed on theinformation display. For example, an OS output handler 715 receives thevisual output from the application and decides how much of the entirewindow will be displayed, where on the information display the windowwill be located, and communicates the visual output 783 to theinformation display. The OS output handler also communicates the visualoutput to an OS remote output handler 770. The OS remote output handlercommunicates the visual output to an output handler 720 at the remotedesktop client application at the first user's computer, whichcommunicates the visual output to an OS output handler 714 for displayat an information display 710.

In accordance with an embodiment, any changes made by the first user attheir local application are executed only at the local application. Theresult is visible on the information display, in addition to the remotedesktop window on the second users information display. For example,computer 706 includes the local application and the remote desktopclient application. The remote desktop client application includes aninput handler 716 and the output handler. An interaction input/request780 is received at an OS input handler 712. The input handlercommunicates the input to the local application, which processes theinput and produces a visual output to be displayed on the informationdisplay. The OS output handler receives the visual output from theapplication and decides how much of the entire window will be displayed,where on the information display the window will be located, andcommunicates the visual output 782 to the information display. The OSoutput handler also communicates the visual output to an OS remoteoutput handler 740, which in turn communicates the visual output to theoutput handler at the second user's remote desktop client application.The output handler communicates the visual output to the OS outputhandler for display at the information display for the second user 711.

In accordance with an embodiment, in those instances when the first userdecides to interact with the second user's local application through theremote desktop client application, then an instance of the second user'slocal application will be instantiated on the first user's computer. Thesecond user's information display will show two applications running;the first user's local application and the instance of the second user'slocation application. For example, an input to interact with the seconduser's local application is received at an OS input handler 712, and iscommunicated to an input handler 716 at the remote desktop clientapplication. The input is communicated from the input handler to an OSremote input handler 766, which communicates with the local applicationat the second user's computer to receive current state information forthe local application. The OS remote input handler communicates thecurrent state information to an application launcher 703, and theapplication launcher uses the current state information to instantiatean instance of the second user's local application on the first user'scomputer having substantially the same application state. In accordancewith an embodiment, after an instance of the second user's localapplication has been instantiating on the first user's computer,subsequent interactions between the first user and the second user'slocal application can update the state of the instance of the seconduser's local application on the first user's computer using the currentstate information of the second user's local application. Similarly, inaccordance with an embodiment, after an instance of the second user'slocal application has been instantiating on the first user's computer,subsequent interactions between the second user and that instance of thesecond user's local application on the first user's computer can updatethe state of the second user's local application using the current stateinformation of that instance of the second user's local application onthe first user's computer.

In accordance with an embodiment, in those instances when the seconduser decides to interact with the remote desktop application to interactwith the first user's local application, then an instance of the firstuser's local application will be instantiated on the second user'scomputer. The first user's application display will show twoapplications running; the second user's local application and theinstance of the first user's location application. For example, an inputto interact with the first user's local application 708 is received atthe input handler 726 at the second user's remote desktop clientapplication from the OS input handler, and communicated from the inputhandler to an OS remote input handler 736. The OS remote input handlercommunicates with the local application 708 at the first computer toreceive current state information associated with the local application,and communicates the current state information to an applicationlauncher 704. The application launcher uses the current stateinformation to instantiate an instance of the first user's localapplication having substantially the same application state. Inaccordance with an embodiment, after an instance of the first user'slocal application has been instantiating on the second user's computer,subsequent interactions between the second user and the first user'slocal application can update the state of the instance of the firstuser's local application on the second user's computer using the currentstate information of the first user's local application. Similarly, inaccordance with an embodiment, after an instance of the first user'slocal application has been instantiating on the second user's computer,subsequent interactions between the first user and that instance of thefirst user's local application on the second user's computer can updatethe state of the first user's local application using the current stateinformation of that instance of the first user's local application onthe second user's computer.

FIG. 8 shows a flow chart of a method in accordance with an embodiment,for providing two-way collaboration in a collaborative desktop system.Although in this example a first user and a second user are described,any plurality of users can collaborate. Additionally, although the userswill be referred to as a first user and a second user, either user canbe an active user or a passive user. At step 802, an application (e.g.,a local application) and a remote desktop client application areprovided at a first computer. The local application can be, e.g., a webbrowser or another application. The remote desktop client application isused by the first user to interact with an application on a secondcomputer. At step 804, an application (e.g., a local application) and aremote desktop client application are provided to a second user. At step806, the first user interacts with the local application at the seconduser's computer. The first user interacts with the second user's localapplication using the first user's remote desktop application. At step808, an application instance of the second user's application isinstantiated on the first user's computer after the first user interactswith the second user's local application. At step 810, the current stateof the application instance of the second user's application is adjustedto the same state as second user's local application. At step 812, thesecond user interacts with the first user's local application using thesecond user's remote desktop client application. At step 814, anapplication instance of the first user's application is instantiated onthe second user's computer after the second user interacts with thefirst user's local application. At step 816, the current state of theapplication instance of the first user's application is adjusted on thesecond user's computer to the same state as first user's localapplication.

In accordance with various other embodiments, the remote desktop clientapplication used by the second user can display the name and/or icon ofthe application(s) running on the first user's computer. When the seconduser requests to activate one of these applications by selecting theapplication name or icon of the available applications, the local startand activation sequence taken on the second user's computer toinstantiate a local instance of the first user's application will onlylaunch, activate, and bring the application to the same state locally onthe second user's computer. Since there are no local actions made by thesecond user through the remote desktop system, no actions are taken onthe new local application. In accordance with various other embodiments,the remote desktop client application used by the second user candisplay an indication of the first user, such as a name indicator, colorindicator or icon indicator representing the first user. When the seconduser requests to activate an application on the first user's device, thecurrently active application on the first user's device is instantiatedto the same state locally on the second user's device. For example,devices with smaller screens, such as mobile phones or other portablemobile devices, where there is not enough screen size to show the entirewindow of an application, but there is enough space on the display toshow a list of available applications, or an indication of the userhosting the application, can be used to select an application running ona first computer or be used to select a user hosting the application.This provides for the ability to transfer applications with statebetween both mobile and mobile devices or mobile and standard devices.

The above examples illustrate how two or more users can collaborateusing a collaborative desktop system. Users of a shared application caninteract with the application without impacting the other users. Forexample, the interaction made by a remote user will result in a localapplication being opened on the remote user's computer. The applicationis opened in the same state as the shared application, and anyinteraction on the local application is executed locally on the remoteuser's computer. Since all information displays are shared between allusers, the other users will still see the remote desktop window showingthe shared application, where the shared application and state isunchanged. The other users will also see a new remote desktop windowfrom the new remote active user. The new remote desktop window willdisplay the same application as on the shared application, but in adifferent state.

In accordance with an embodiment, systems such as those described hereincan significantly impact the sales and advertisement of applications.For example, presenting an application to a user that has a need for theapplication is a key issue when dealing with application sales. Industryexperience places significant weight on reviews by other users as animportant factor in deciding whether to purchase an application, andeven more valuable is the word-of-mouth from trusted users such asfriends and family. However, a user may not be aware of whatapplications a friend or family member is using, so it is left to chanceif the name of application is mentioned in conversation. Oftentimes,even when the name of the application is known, the user still needs tofigure out how and where to locate the application, and how to downloadthe program. This is often accomplished by the user searching on theinternet, or they will search in a dedicated application store (forexample Amazon or Apple's AppStore etc.). Nonetheless, there are severallayers of friction to discovery and distribution of applications basedon the most valuable lead: family and friends.

Systems such as those described herein can also aid in applicationdiscovery and distribution. For example, if one of the users (e.g., ahost user) is using an application that another connected user (e.g.,client/remote user) finds interesting, then the connected user caninitiate an interaction with the host application through a remotedesktop window. Assuming the application is not yet installed on therequesting user's computer, then the host computer can inform therequesting user's computer about where and how to download and installthe application. In the event the application is free, the installationcan takes place on the requesting user's computer, after which theapplication will start in same state as on the host computer, but nowlocally on the requesting user's computer. If the application is apremium application, a purchase prompt can be displayed to purchase theapplication.

Example Use Cases

In accordance with various embodiments, the system can be implemented tosupport a variety of different use cases and applications. Some examplesof these use cases and applications are described below by way ofillustration. It will be evident that embodiments of the invention canbe implemented to support other use cases and applications, and that theinvention is not limited to the example provided herein.

For example, in accordance with an embodiment, one or more users canoperate their own computer, e.g., either side-by-side or remote from oneanother. The information display of each user can have a window of theother users' current activities. Each user can perform localinteractions with the applications of the other users through the remotedesktop window. In accordance with an embodiment, one or more users canuse one large display, or several smaller display devices that are allconnected to the same computer. Each user will have their owninformation display mapped to the display device (e.g., a split-screensetup). In accordance with an embodiment, each information display canhave a remote desktop window from each of the other active informationdisplays to allow the users to start local activities on their owninformation display. For example, an interactive large screen TV can besetup where the users can split the screen into two informationdisplays. Each information display can render the applications run bythe user owning that information display, in addition to a remotedesktop window of the other users information display. The input made byusers onto the large display can be accomplished through remote controls(e.g. mobile phones or similar smart electronic devices, but is notlimited thereto).

In accordance with an embodiment, systems described herein can beimplemented in two ways, a preemptive-feed system or a request-responsesystem. The preemptive-feed system can bundle state information with avisual output used at an information display. This allows connectedclients to the system to launch local applications at any time withoutrequesting more data from a host system having the applications. Thebenefit of this approach is faster application load time. In accordancewith an embodiment, the request-response system (as illustrated in FIG.5 and FIG. 7) only forwards the visual output to be displayed on aninformation display of a remote user.

In accordance with an embodiment, a number of privacy options can beimplemented. For example, a host user can prevent other users (e.g.,remote users) from interacting with a host application and/or viewingthe contents of the host application. Additionally or alternatively, apermission request system can be implemented where the host userreceives a request to allow remote users to view and/or copy the hostapplication's current state. In accordance with an embodiment, adecision to automatically share an application can be based on theapplication kind, and/or what the host user is using the applicationfor. For example, remote users can be automatically restricted fromcopying the host users browsing session that is based on SSL encryption.Such a session can be, e.g., a banking or email session.

Shared Desktop Implementation

FIGS. 9A-9D shows an illustration of how the system can be used toprovide a shared desktop implementation, in accordance with anembodiment. As shown in FIGS. 9A-9D, a first user can use, e.g., amulti-touch enabled desktop computer 904 with a browser window 906, tobrowse the Internet or web within a browsing session. A second user, whois aware of and wants to participate in the browsing session, startingfrom a substantially similar position and state as the first user, canissue a split-screen or other recognized gesture or command to instructthe system to split the screen display.

For example, in accordance with an embodiment either the first or seconduser can instruct the system 902 to split the screen display by tracingtheir finger from a border of the display towards its center. Inaccordance with an embodiment, while the user traces their finger thecomputer graphically illustrates the movement by displaying a colored“T”-shaped graphical device 908 that grows in size as the gestureapproaches activation. When the user's finger travels a sufficientdistance, the system recognizes this as a request to split the screendisplay. In accordance with other embodiments, the system can recognizeother gestures or commands to split the screen display, e.g., by use ofan explicit button or voice command.

In accordance with an embodiment, when the system detects a request tosplit the screen display, a new browser window is created and displayed912, which is designated to the second user, but which uses the sameuniform resource locator (URL), content, zoom and scroll bar position914, as the original browser window, which remains designated to thefirst user. In other words, the new browser window and browsing sessionincludes substantially the same position and state as the originalbrowser window and browsing session. Additional browser windows andbrowsing sessions can be similarly created, each one includingsubstantially the same position and state as its predecessor.

In accordance with an embodiment, when multiple users are participatingand browsing using a split-screen display, each user is associated witha color or other indication that uniquely identifies that particularuser. Each browser window is associated with a colored border 916, 918and 932 that identifies its controlling user, and also displays one ormore appropriately colored msync icons 920, 924 for each of the otherparticipating users. Colors can be assigned using, e.g., a static list,wherein the first entry in the list is always assigned to the first userof the system, and each subsequent request to split the screen displayincrements the color index and assigns the next color in the list to thenext participating user. Different colors can also be used during theoperation of the split gesture, e.g., the sides (left, right, top,bottom) of the “T”-shaped graphical device can be given a user color forwhich side of “T” that the new user window will be created. Should theuser choose to finalize the split gesture, the side chosen to create thenew window will be determined from the split gestures position relativeto the center of the original user's window—any of the 4 borders of themonitor can be used to active a split.

Subsequent to the system splitting the display screen, and creating anddisplaying the new browser window, the second user can thereaftercontrol their instance of the browser window and browsing session tobrowse the web, e.g., by clicking on links therein to other articles, orotherwise navigating away from the initial page or scroll the content.

At any subsequent point in time, if any user wants to “borrow from” thebrowsing session of any other user (e.g., because they are interested inthe contents of that other user's current browser window) the user canperform an msync with that user by clicking the appropriately coloredmsync icon 936. Such borrowing can happen back-and-forth between themultiple users.

The above example illustrates how two or more users can browseindependently, and yet synchronize their browsing location with eachother at any time once a publishing session has started. Additionalusers can be added by performing a split gesture again. Because themulti-user behavior is taking place on the same monitor (or in severalmonitors in close proximity), a publishing group can be created withoutprior account creation or authentication, which simplifies the processfor the user of getting started. Additionally, in accordance withvarious other embodiments, a publishing group can be automaticallydefined based on users connecting within the same network, e.g., an IPsubnet. This setup allows mobile users connected on the same network tosynchronize their browsing location with each other, and follow othermobile users, as well as all desktop users on the same network.

Shared Group Implementation

FIGS. 10A-10C shows an illustration of how the system can be used toprovide a shared group implementation, in accordance with an embodiment.As shown in FIGS. 10A-10C, a group of two or more users are viewingcontent on a shared viewing device 1002 that includes a display 1004,e.g., watching a movie on a television together. At least one or more ofthe group 1010 is also currently browsing the web on a personal Internetor web-enabled device 1006, e.g., a smartphone, tablet, or othercomputing device that includes a display and a browser window.

In accordance with an embodiment, each user and/or their web-enableddevice is associated with a color or other indication that uniquelyidentifies that particular user/device within the group. When theirbrowser recognizes other available viewing devices or web-enableddevices, it displays an appropriately-colored icon for those otherdevices 1036. Colors can be assigned using, e.g., a static list, whereinthe first entry in the list is always assigned to the first user/device,and each subsequent user/device is assigned the next color in the list.In accordance with an embodiment, a shared viewing device push icon 1016can be provided to allow a user to push a copy of their online contentto the shared viewing device. A publish icon 1014 can be provided toallow a user to publish a live feed of their online content to otherInternet-enabled devices within the group. When the recipient device isthe shared viewing device, e.g., a television, the color associated withthe device that pushed its online content to the television can be usedas a border around the television window. This allows each user viewingthe television to know which user's online content is currently beingdisplayed on the television. While television is mentioned in thisscenario, it could also be any other device used for media presentationsuch as a computing device with a large monitor or a projector.

When a device is used to publish a copy of their online content to otherweb-enabled devices, any user currently participating in the group 1040can subscribe to or otherwise copy the current state of online contentfrom any publishing user within the group, regardless of whether thatcontent is currently being displayed on the shared viewing device. Forexample, when group members are physically present within the same room,a wish to synchronize devices can be conveyed verbally. This means that,although the shared viewing device is a useful means of informing usersof each other's browsing session, the system can also operate without ashared viewing device, relying instead on other communication (e.g.,verbal) means which leads to users pressing a msync button 1036 on theircomputing device 1006.

When a user's online content is msync'd to a subscribing web-enableddevice, the state of their browsing session is copied to the subscribingdevice, and the browser window on the subscribing device is updated touse the same uniform resource locator (URL), content, zoom and scrollbar position, as the original browser window, which remains designatedto the publishing device. In other words, the new browser window andbrowsing session includes substantially the same position and state asthe original browser window and browsing session.

In accordance with an embodiment, if a user clicks their publish icon,their publish icon is modified to indicate the user is in a publishingmode, and that other users within the group can copy their current stateof online content. If the user also clicks their push icon, the onlinecontent currently displayed on their device is also replicated live tothe shared viewing screen 1002, surrounded by the user's associatedcolor.

Depending on the implementation, e.g., the available display area on theshared viewing device and the settings of the users, the system cansimultaneously display a plurality of content windows on the sharedviewing device. When no users are pushing content, the default is to usethe entire screen for display. As users begin pushing content, thescreen display can be

split, up to a maximum determined limit, at which point the system canbeing removing the oldest content first. This provides a flexibleapproach for allowing additional users to participate in the group whileusing the shared display real estate effectively.

At any time a user can either discontinue publishing and/or discontinuepushing their online content 1020. Discontinuing pushing does not initself stop the publishing, but does result in that users/device'swindow being removed from the shared display, and leaves room for otherusers to push their content, or for shared content to assume a largerportion of the available area. If the user discontinues publishing, thenthe system considers the user to have left the group, and willdiscontinue any live online content feed being delivered to that device.

In accordance with an embodiment, when a new user 1032 joins the groupand starts a compatible browser device 1030, by default the new userwill not be in a publishing mode, but instead a publish icon 1038 willbe available. If a compatible shared viewing device is present then ashared viewing device push icon will also be available in the new user'sassigned color. The operation of these icons is as described above. Foreach user that is currently publishing content within the group, auser-colored msync button will be available. Since the new user will notyet be publishing, other users will not yet see an msync button thatallows them to subscribe to or otherwise copy the current state ofonline content from that new user. Once the new user choose to push orpublish their online content then, in addition to the previouslydescribed actions, all other users within the group will now see auser-colored sync button 1036 for that new user.

In accordance with an embodiment, the system can use a cloud-based orother Internet-based service to connect groups and devices, and toenable sharing of online content or browsing sessions over the Internet.In accordance with other embodiment, groups and devices can beautomatically created, e.g. within a shared IP subnet, or within ashared WiFi zone, which minimizes the need for configuring the system.

The above example illustrates how two or more personal Internet orweb-enabled devices can be used to independently browse content, whileat the same time synchronize with each other when needed, andillustrates how a shared viewing device is a useful means of informingusers of each other's browsing session, although as described above thesystem can also operate without a shared viewing device, relying insteadon other communication (e.g., verbal) means to trigger the interest tostart publishing and to perform an msync.

The present invention may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the present invention includes a computer programproduct which is a storage medium or computer readable medium (media)having instructions stored thereon/in which can be used to program acomputer to perform any of the processes of the present invention. Thestorage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, microdrive, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flashmemory devices, magnetic or optical cards, nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data.

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

I claim:
 1. A computing system architecture that enables a group ofusers to share information, comprising: a first computing deviceassociated with a first user, the first computing device having a firstcomputing display; a second computing device associated with a seconduser, the second computing device having a second computing display; thefirst computing device and the second computing device each having adesktop logic, the desktop logic is enabled to allow the first computingdevice to define a first group, allow the second computing device torequest to join the first group, and upon the second computing devicejoining the first group, the first computing device and the secondcomputing device exchange indicators where each indicator uniquelyidentifies each computing device, the first computing display presentingat least a first window that displays a first application in a firststate, and the second computing display presenting at least a secondwindow that displays a second application in a second state, eachapplication is capable of sending and receiving information regarding astate of each window being displayed by its respective computingdisplay; and accepting, at the first computing device, information fromthe second computing device, the information comprising an areainformation, a layer information, and a window identificationinformation for the second window.